Policy Evaluation and Temporal-Difference Learning in Continuous Time and Space: A Martingale Approach

We propose a unified framework to study policy evaluation (PE) and the associated temporal difference (TD) methods for reinforcement learning in continuous time space. show that PE is equivalent maintaining martingale condition of process. From this perspective, we find mean--square TD error approximates quadratic variation thus not suitable objective PE. present two use characterization designing algorithms. The first one minimizes ``martingale loss function, whose solution proved be best approximation true value function sense. This method interprets classical gradient Monte-Carlo algorithm. second based on system equations called orthogonality conditions with ``test functions''. Solving these different ways recovers various algorithms, such as TD($\lambda$), LSTD, GTD. Different choices test functions determine what sense resulting solutions approximate function. Moreover, prove any convergent time-discretized algorithm converges its continuous-time counterpart mesh size goes zero. demonstrate theoretical results corresponding algorithms numerical experiments applications.